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Abstract 


Schedule-  or  event-driven  reviews  are  a  crucial  element  of  any  major  software  development 
project.  Such  reviews  tend  to  focus  on  different  aspects  of  development,  and  different  types  of 
reviews  provide  different  benefits.  The  sum  of  these  reviews,  however,  is  inadequate  to  address 
the  needs  of  software  development  in  a  complex  system  of  systems  (SoS)  environment.  What  is 
needed  is  a  true,  evidence -driven,  SoS-level  evaluation  capable  of  providing  an  overall  assess¬ 
ment  of,  and  insight  into,  the  software  development  effort  in  that  context. 

This  report  discusses  the  application  of  the  Lifecycle  Architecture  (LCA)  event  to  what  was  an 
enormously  complex  SoS  program:  the  Army’s  Future  Combat  Systems.  From  the  FCS  expe¬ 
rience,  readers  will  gain  insight  into  the  issues  of  applying  the  LCA  in  an  SoS  context  and  be 
ready  to  apply  the  lessons  learned  in  their  own  domains. 
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1  Introduction 


Schedule-  or  event-driven  reviews  are  a  crucial  element  of  any  major  software  development 
project.  Such  reviews  tend  to  focus  on  different  aspects  of  development:  producihility  reviews 
focus  on  the  ability  to  produce  the  software  within  available  resources;  capability  reviews  focus 
on  the  services  being  provided  by  the  software;  integration  and  test  reviews  focus  on  the  readiness 
of  the  software  to  enter  or  transition  between  those  phases  of  the  development  life  cycle;  schedule 
reviews  focus  on  the  development  effort’s  adherence  to  planned  timelines;  and  so  on.  Most  over- 
all-systems  reviews  focus  mainly  on  the  system-artifact  functional  definitions,  and  relatively 
lightly  on  evidence  that  the  artifacts  described  will  meet  the  system’s  key  performance  parameter 
requirements. 

These  different  types  of  reviews  provide  different  benefits  and,  taken  together,  they  provide  valu¬ 
able  insights  into  the  software  development  project  at  various  stages  of  the  life  cycle.  The  sum  of 
these  reviews,  however,  is  inadequate  to  address  the  needs  of  software  development  in  a  complex 
system  of  systems  (SoS)  environment,  which  is  a  significant  difficulty  since  systems  of  systems 
are  becoming  almost  de  rigueur  solutions  in  U.S.  defense  acquisition  programs.  In  order  to  deliver 
orders  of  magnitude  improvement  in  capabilities  for  the  Warfighter,  defense  programs  increasing¬ 
ly  are  weaving  together  existing  systems  into  systems  of  systems  (what  is  sometimes  called  an 
acknowledged  SoS)  or  building  a  new  SoS  from  scratch  (a  directed  SoS).*  While  systems  of  sys¬ 
tems  can  deliver  unprecedented  capability,  they  also  present  a  number  of  technical  and  manage¬ 
ment  challenges. 

The  notion  of  an  SoS  is  an  abstract  concept,  often  without  a  physical  manifestation  that  represents 
the  cognitive  whole,  in  which  a  collection  of  systems  taken  together  and  operating  in  concert  with 
each  other  provides  greater  functionality  than  the  sum  of  the  parts.  The  dynamic  interaction  of  the 
constituent  pieces  yields  emergent  behavior.  Thus,  an  SoS  adds  complexity  to  the  typical  acquisi¬ 
tion  program  by  introducing  the  need  to  manage  many  pieces,  each  a  complex  system  in  its  own 
right,  in  addition  to  a  somewhat  abstract  end  product. 

One  of  the  difficulties  with  an  SoS  approach  is  that  while  interfaces  may  be  known  in  advance, 
how  those  interfaces  may  be  used  to  deliver  a  given  capability  within  the  SoS  construct  under  any 
given  set  of  conditions  is  not  necessarily  known.  For  example,  in  a  networked  fires  scenario,  a 
sensor  may  detect  and  report  a  target,  and  a  set  of  algorithms  may  determine  which  of  several 
available  shooters  is  best  able  to  engage  the  target;  it  is  not  known  in  advance  which  shooter  will 
be  selected.  This  emergence  of  behavior  makes  it  difficult  to  have  confidence  in  one's  understand¬ 
ing  of  the  contribution  of  any  component-level  design  to  the  higher  aggregation;  small,  seemingly 
non-critical  changes  can  have  subtle  and  unrecognized  impacts  to  other  components  or  to  overall 
SoS  performance. 


The  related  taxonomy  of  SoS  was  originated  in  a  report  by  Mark  Maier  [Maier  1999]  and  extended  to  include 
Acknowledged  SoS  in  a  recent  CrossTa//c  article  [Dahmann  2008].  A  summary  may  be  found  in  a  2009  SEI 
report  [Bergey  2009]. 

For  thorough  discussions  of  emergent  behavior,  see  related  reports  by  John  Flolland  [Flolland  1998]  and  David 
Fisher  [Fisher  2006]. 
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Thus,  traditional  software  review  approaches  fall  well  short  when  used  as  a  means  for  understand¬ 
ing  and  evaluating  software  at  an  SoS  level.  What  is  needed,  then,  is  a  method  for  evaluating 
software  at  the  SoS  level,  in  addition  to  the  more  traditional  means  of  reviewing  component-level 
software  designs.  What  is  needed  is  a  true  SoS-level  evaluation  capable  of  providing  an  overall 
assessment  of,  and  insight  into,  the  software  development  effort  in  that  context — one  that  can 
provide  a  review/assessment  of  how  the  developed  software  capability  enables  the  program’s  de¬ 
sired  operational  capability. 

The  Future  Combat  Systems  program,  which  was  a  very  large  and  complex  SoS  development  ef¬ 
fort  undertaken  by  the  U.S.  Army,  faced  just  such  a  challenge,  and  hypothesized  that  something 
akin  to  a  Lifecycle  Architecture  (LCA)  milestone  conducted  at  the  SoS  level  might  be  the  answer. 
According  to  the  Rational  Unified  Process,  the  LCA  marks  the  conclusion  of  the  elaboration 
phase  of  software  development,  when  the  requirements  baseline  is  set  and  the  architecture  is  com¬ 
plete  [Kroll  2003,  Boehm  1996].  As  extended  to  systems  engineering  in  works  by  Richard  Pew 
[Pew  2007]  and  Barry  Boehm  [Boehm  2007],  the  goal  of  anchoring  events  such  as  the  LCA  is  to 
assess  program  risk  by  examining  evidence  provided  by  the  developers  that  a  system  developed  to 
the  architecture  can  satisfy  the  requirements.  This  helps  to  ensure  that  program  goals  now  and  in 
the  future  can  be  satisfied.  The  LCA,  in  particular,  strives  to  ensure  that  a  system,  as  architected, 
can  be  constructed  within  planned  cost  and  schedule. 

Applying  the  LCA  approach  to  a  system  of  systems  introduces  some  interesting  challenges;  just 
as  an  SoS  is  not  merely  a  roll-up  of  its  constituent  systems,  so  it  is  that  an  SoS-level  LCA  is  more 
than  a  roll-up  of  lower  level  LCA  anchor  points.  A  roll-up  approach  would  fail  to  capture  all  of 
the  interesting  interactions  that  underpin  the  SoS.  Further,  the  nature  of  SoS  programs  causes  dif¬ 
ferent  elements  to  be  at  different  stages  of  development  at  any  given  time.  Therefore,  the  notion 
that,  at  the  SoS  level,  the  architecture  is  complete  (and,  implicitly,  that  further  development  has 
not  yet  begun)  is  unsound. 

This  report  looks  at  the  adaption  and  application  of  the  Lifecycle  Architecture  milestone  to  the 
software  and  computing  elements  of  the  former  Future  Combat  Systems  program.  Section  2  pro¬ 
vides  some  background  information  about  the  LCA  anchor  point  and  the  FCS  program  to  set  the 
context.  Sections  3  and  4  discuss  the  LCA  anchor  point  as  it  was  developed  and  applied  to  FCS. 
Section  5  discusses  general  outcomes  of  the  LCA  approach.  Section  6  reviews  some  lessons 
learned  from  the  experience,  and  Section  7  offers  some  conclusions.  From  the  FCS  experience, 
readers  will  gain  insight  into  the  issues  of  applying  the  LCA  in  an  SoS  context  and  be  ready  to 
apply  the  lessons  learned  in  their  own  domains. 
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2  Background 


2.1  The  Lifecycle  Architecture  Milestone 

The  LCA  milestone  represents  a  point  in  a  program  where  stakeholders  come  together  to  assess 
the  work  that  has  been  completed  through  the  elaboration  phase  (the  phase  in  which  requirements 
and  architecture  models  are  largely  completed)  and  evaluate  the  risks  of  moving  forward  into  the 
construction  and  transition  phases  (the  phases  in  which  design,  implementation,  and  test¬ 
ing/validation  are  performed).  The  name  derives  from  the  event-oriented  nature  of  the  review:  it  is 
a  life-cycle  look-ahead,  conducted  at  a  point  where  the  architecture  is  sufficiently  mature  to  allow 
reasoning  about  the  relative  risks  of  continuing  to  the  construction  phase  of  development.  The 
LCA  differs  from  traditional  milestone  reviews  such  as  preliminary  design  reviews  (PDRs)  and 
critical  design  reviews  (CDRs),  which  tend  to  focus  superficially  on  voluminous  system  descrip¬ 
tion  data,  in  that  the  LCA  is  a  risk-based  assessment  focused  on  the  feasibility  of  proceeding  with 
additional  work. 

The  key  to  the  LCA  is  the  feasibility  rationale,  which  documents  evidence  provided  by  the  devel¬ 
opers  that  the  proposed  architecture  can  be  implemented  to  satisfy  its  requirements  within  the  de¬ 
fined  schedule  and  project  budget.  The  evidence  should  be  objective  and  consistent  within  itself  to 
justify  the  confidence  of  all  stakeholders  in  moving  forward  into  the  latter  phases  of  the  develop¬ 
ment  life  cycle.  Thus,  it  is  important  to  understand  that  the  LCA  is  not  simply  a  technical  assess¬ 
ment,  but  a  programmatic  one.  Successful  completion  of  the  LCA  anchor  point  represents  a  com¬ 
mitment  by  all  stakeholders  to  proceed  with  the  program,  based  on  objective  evidence  that  the 
risks  of  moving  forward  have  been  identified  and  sufficiently  mitigated  to  ensure  a  reasonable 
chance  of  success. 

The  LCA  feasibility  rationale  is  relatively  straightforward  in  the  context  of  a  traditional  system 
development  program.  For  a  complex  system  of  systems  program,  the  feasibility  package  be¬ 
comes  less  clear.  Simply  rolling  up  results  from  the  LCA  anchor  points  of  constituent  systems  is 
not  a  sufficient  approach  because  important  inter-system  and  SoS  implications  might  easily  be 
missed.  The  feasibility  of  executing  each  individual  software  package/system  does  not  necessarily 
correlate  to  the  feasibility  of  executing  the  entire  SoS. 

2.2  Future  Combat  Systems 

The  FCS  program  was  envisioned  by  the  U.S.  Army  as  a  family  of  manned  and  unmanned  wea- 

'1 

pon  systems,  linked  by  a  sophisticated,  wireless,  battlefield  network  (see  Figure  1).  Additionally, 
the  FCS  platforms  would  have  been  interoperable  with  currently  fielded  weapon  systems,  creating 
a  future  fighting  force  with  unprecedented  and  unparalleled  situational  awareness  and  understand¬ 
ing,  operational  synchronization,  and  connectivity  throughout  the  joint  battlespace  [PM  FCS 
2008].  This  requirement  to  be  part  of  a  larger  force  also  created  a  major  challenge  for  FCS  to 
maintain  interoperability  with  numerous  independently  evolving  external  systems. 


In  June  2009,  the  Honorable  Ashton  Carter,  undersecretary  of  defense  for  acquisition,  technoiogy,  and  iogis- 
tics,  directed  the  canceiation  of  the  FCS  program  and  its  restructure  into  smaiier  deveiopment  projects. 
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Unmanned  Air  &  Ground  Systems  (6) 


Manned  Ground  Vehicles  (8) 


Figure  1 :  FCS  System  of  Systems  [FCS  2008a] 

As  shown  in  Figure  2,  FCS  software  is  pervasive  throughout  the  SoS,  providing  a  host  of  battle 
command  applications  to  the  vehicle  platforms  at  the  highest  level,  and,  through  the  FCS  System 
of  Systems  Common  Operating  Environment  (SOSCOE)  software,  tying  those  vehicles  to  exter¬ 
nal  sensors  and  to  lower  network  communications  services  for  accessing  the  Global  Information 
Grid  (GIG). 

ECS  was  executed  by  a  Lead  Systems  Integrator  (LSI)  team  composed  of  the  Boeing  Company 
and  Science  Applications  International  Corporation  (SAIC).  The  LSI  assembled  a  best-of-industry 
team  of  prime  contractors  and  suppliers,  collectively  referred  to  as  One  Team  Partners  (OTPs). 
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Figure  2:  Software  is  a  Key  Enabier  of  the  FCS  Concept  and  the  Globat  Information  Grid  [FCS  2008b] 

While  the  core  FCS  program  focused  on  producing  its  main  fourteen  systems  and  the  network  to 
tie  them  together,  the  program  also  established  a  technology  bridge  to  the  Army’s  Current  Force. 
Shown  in  Figure  3,  as  FCS  technologies  matured,  they  were  planned  for  introduction  to  the  Cur¬ 
rent  Force  through  a  series  of  spin  outs,  allowing  the  Current  Force  to  become  increasingly  capa¬ 
ble  as  it  evolved  toward  the  Future  Force  and  also  ensuring  it  would  be  interoperable  with  new 
FCS  systems  as  they  are  fielded. 
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Figure  3:  FCS  Core  Program  and  Spin  Outs 

The  complexity  of  the  program  clearly  demonstrated  the  need  for  ensuring  SoS-level  stakeholder 
commitment  to  moving  forward  at  key  development  milestones. 

Software  for  the  program  was  developed  in  a  series  of  incremental  huilds,  each  with  its  own  set  of 
review  events  having  a  specific  area  of  concern: 

•  Evaluation  of  functionality  to  be  developed  against  cost  and  schedule  resources 

•  Evaluation  of  delivery,  integration,  and  test  timelines  and  criteria 

•  Evaluation  of  requirements  and  interface  maturity 

•  Evaluation  of  horizontal  and  vertical  integration  of  capabilities  and  requirements 

Eor  each  software  build,  two  levels  of  review  events  existed,  one  level  for  individual  software 
packages  and  another  for  integrated  builds. 

Although  EGAs  were  conducted  regularly  at  constituent  systems  levels,  the  focus  of  those  events 
was  necessarily,  and  appropriately,  build-specific  and  functionality-specific.  The  existing  reviews, 
based  almost  entirely  on  artifacts,  provided  important  insight  into  the  development  plans  and  re¬ 
sults  of  individual  software  builds,  but  tended  to  focus  on  evaluating  design,  architecture,  and  re¬ 
quirements  to  develop  a  solution  that  should  meet  program  needs;  comparatively  little  emphasis 
was  placed  on  modeling,  simulation,  and  test  evidence  to  show  whether  the  resultant  software 
would  meet  program  needs.  None  of  the  events  provided  insight  into  the  entire  ECS  software  ef¬ 
fort  or  how  that  effort  offered  a  demonstrated  contribution  to  the  operational  needs.  At  the  SoS 
level,  a  broader  view  was  called  for;  one  that  considered  cross-system  functionality  across  mul- 
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tiple  builds  to  develop  confidence  that  the  desired  end  state  would  be  both  reachable  and  satisfac¬ 
tory. 

Although  the  need  for  an  LCA-like  anchor  point  at  the  SoS  level  was  apparent,  and  the  LSI’s 
Software  Development  Plan  included  a  Data  Item  Description  for  a  Feasibility  Rationale  docu¬ 
ment,  it  was  not  clear  to  what  extent  this  applied  contractually  across  the  program.  As  it  had  many 
times  before,  the  FCS  program  broke  new  ground,  this  time  by  defining  its  SoS  LCA  procedures. 
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3  Defining  the  SoS  LCA  Evaiuation  Process 


The  first  step  in  the  process  for  defining  an  SoS-level  LCA  anchor  point  was  to  set  up  a  team  of 
experts  from  academia,  hereinafter  referred  to  as  the  SoS  LCA  Team,  with  representatives  from 
the  Army  providing  guidance  and  representatives  from  the  LSI  facilitating  access  to  program  arti¬ 
facts  and  personnel.  Team  members  brought  a  range  of  programmatic  and  technical  expertise  to 
the  effort,  including  a  deep  understanding  of  the  typical  LCA  process.  They  also  brought  the  de¬ 
gree  of  independence  necessary  to  assure  both  the  LSI  and  the  Army  of  an  unbiased  result. 

As  mentioned  earlier,  the  PCS  program  had  been  executing  LCAs  at  the  supplier  and  first-tier 
integration  levels,  so  there  was  no  need  to  start  from  scratch.  However,  simply  rolling-up  the  re¬ 
sults  of  these  lower  level  reviews  would  invite  missing  critical  subtleties  in  the  cross-system  rela¬ 
tionships.  For  example,  a  relatively  low-risk  design  change  in  a  battle  command  system  might 
have  unforeseen  implications  for  a  logistics  system.  In  addition,  the  lower  level  LCA  reviews 
were  focused  on  specific  builds,  often  at  different  points  in  time,  whereas  the  SoS-level  LCA  was 
obliged  to  project  across  builds.  Further,  other  FCS  software  reviews  focused  on  the  plans  for  the 
immediately  upcoming  phase  of  a  given  build.  The  SoS  LCA  had  to  consider  existing  data  and 
results  from  prior  builds  as  well.  Thus,  it  was  necessary  to  construct  an  evaluation  process  that 
was  anchored  by  the  best-available  facts  and  data  as  the  basis  for  projecting  forward. 

3.1  Conceptualizing  the  SoS  LCA 

From  the  outset,  the  SoS  LCA  definition  effort  was  steered  by  few  key  guidelines  that  helped 
drive  the  ultimate  solution.  These  guidelines  were: 

•  Answer  the  following  questions: 

Can  what  is  planned  to  be  built  really  be  built? 

Will  the  various  pieces  add  up  to  the  desired  whole? 

Are  the  risks  clearly  identified? 

•  Base  answers  on  evidence  rather  than  opinion. 

•  Discover  issues  so  that  they  can  be  fixed  sooner  rather  than  later. 

•  Build  the  confidence  needed  in  the  software/computing  system  development  plans  for  a  suc¬ 
cessful  Milestone  decision. 

Without  waiting  for  tests  on  the  final  implementation,  the  SoS  LCA  sought  to  determine  if  FCS 
software  could  be  built  and  if  it  would  satisfy  program  needs.  The  basis  of  these  determinations 
was  objective  evidence:  engineering  work  products  were  evaluated  for  completeness  and  adequa¬ 
cy,  as  well  as  for  consistency  among  and  between  other  work  products  and  plans.  The  SoS  LCA 
focused  on  test  data  and  results  (as  well  as  software  producibility  analysis)  to  evaluate  the  current 
capability  of  FCS  software  and  project  the  ability  to  meet  program  needs. 


Milestone  C  (MS  C)  is  a  capstone  review  in  the  DoD  system  acquisition  life  cycle.  Success  marks  the  comple¬ 
tion  of  the  System  Development  and  Demonstration  (SDD)  phase  of  a  program  and  approves  entry  into  the 
Production  and  Deployment  (P&D)  phase  [DAD  2005]. 
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Rather  than  finding  fault  and  assigning  hlame,  the  goal  of  the  LCA  was  to  surface  problems  as 
early  as  possible  so  that  they  could  be  fixed  at  minimum  cost.  Equally  important  for  ECS  was  the 
building  of  confidence  in  the  software  and  computing  system  development  plans,  since  an  entire 
build  of  software  would  not  yet  have  been  developed  at  the  time  of  the  program’s  planned  Miles¬ 
tone  C  review. 

As  shown  in  Eigure  4,  the  ECS  SoS  LCA  extrapolated,  from  what  was  known  through  what  was 
expected  and  planned,  the  likelihood  of  reaching  the  desired  endpoint  within  a  tolerable  level  of 
risk.  In  the  case  of  ECS,  the  desired  endpoint  was  an  acceptable  implementation  of  Software  Build 
3  Einal  (B3E)  to  support  a  successful  Milestone  C  decision.  The  SoS  LCA  results  were  a  key 
feeder  into  the  program  PDR  event. 


SoS  LCA  extrapolates 
from  facts,  data,  and 
artifacts  to  answer 
the  question: 

“Will  you  get  there?” 
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Figure  4:  SoS  LCA  Determines  the  Liketihood  of  Achieving  an  Acceptable  Outcome 

The  basis  of  the  extrapolation  was  facts  and  data  in  addition  to  other  artifacts.  Eacts  took  the  form 
of  test  results  and  demonstration  outcomes  for  software  that  already  had  been  built.  Data  consisted 
of  simulation  results  that  predicted  operational  performance  of  planned  designs  as  well  as  the  re¬ 
sults  of  various  technical  assessments.  The  remaining  artifacts  of  interest  included  the  various 
plans,  designs,  schedules,  etc.  that  formed  the  basis  of  the  work  yet  to  be  done. 

3.2  Focus  Areas 

The  SoS  LCA  Team’s  first  challenge  was  to  grapple  with  scope.  On  a  very  large  SoS  program 
such  as  ECS,  the  scope  of  the  LCA  becomes  extremely  important.  There  is  no  practical  way  to 
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review  all  of  the  data  one  might  wish  to  analyze  in  order  to  draw  conclusions.  Such  a  detailed  ef¬ 
fort  might  require  thousands  of  labor  hours  spread  over  many  months.  While  the  cost  alone  would 
be  a  significant  deterrent,  the  time  factor  also  would  be  of  concern.  The  longer  it  takes  to  com¬ 
plete  the  examination  of  all  the  evidence,  the  higher  the  risk  that  analyses  performed  early  in  the 
process  will  become  invalid;  as  development  work  continues,  prior  problems  or  unknowns  be¬ 
come  resolved  while  new  issues  may  be  discovered. 

On  PCS,  program  management  had  decided  to  limit  the  scope  of  the  SoS  LCA  to  the  software  and 
computer  processing  elements  of  the  program,  but  even  that  limitation  left  the  range  of  investiga¬ 
tion  too  broad.  There  are  practical  limitations  on  how  much  can  be  reasonably  accomplished  at 
the  SoS  level,  and  it  quickly  became  apparent  to  the  SoS  LCA  Team  that  executing  a  traditional 
LCA  at  the  SoS  level,  even  one  constrained  to  the  software  and  computing  system  elements  of  the 
program,  would  not  be  feasible.  A  detailed  understanding  of  the  designs  would  consume  vast  re¬ 
sources  in  terms  of  time,  money,  and  people.  In  addition,  coordinating  an  effort  of  that  magnitude 
would  become  a  project  within  the  project.  Such  an  LCA  would  be  too  large,  too  complex,  and 
too  resource-intensive  to  be  managed  successfully.  Instead,  the  team  had  to  re -envision  what  a 
modest-sized  team  with  limited  resources  could  reasonably  accomplish.  This  realization  naturally 
led  to  a  focus  on  high-payoff  areas  for  the  program.  These  focus  areas  were  not  necessarily  risk 
areas,  but  rather  crucial  areas  to  successful  development  of  the  PCS  software. 

The  original  ten  high-payoff  areas  postulated  by  the  SoS  LCA  Team  based  on  the  Army  sponsors’ 
priorities  and  the  team  members’  knowledge  of  the  program  were: 

1 .  ability  to  meet  schedules  within  and  across  increments 

2.  ability  to  meet  budgets  within  and  across  increments 

3.  ability  to  integrate  across  Integrated  Product  Teams  (IPTs)  and  suppliers  (including  adequa¬ 
cy  of  requirements,  architectural  consistency,  and  so  on) 

4.  ability  to  achieve/maintain  interoperability  among  independently  evolving  systems  (includ¬ 
ing  Current  Porce  platforms) 

5.  ability  to  coordinate  multiple  baselines  for  the  core  program,  spin  outs,  and  experiments 

6.  ability  to  manage  co-dependencies  between  models,  simulations,  and  actual  systems 

7.  feasibility  of  meeting  required  performance  levels  in  selected  areas  (e.g.,  safety,  security, 
etc.) 

8.  maturity  of  technology  considering  scalability  requirements 

9.  supportability  of  software 

10.  adequacy  of  commercial-off-the-shelf  (COTS)  evaluations  and  mitigations 

A  cursory  review  of  the  results  from  lower  level  LCAs  revealed  that  these  focus  areas  were  com¬ 
monly  cited  as  problems.  With  the  focus  areas  thusly  defined,  the  SoS  LCA  Team  turned  its  atten¬ 
tion  to  developing  the  SoS  LCA  process. 

3.3  The  Initial  Process  Model 

Having  established  the  scope  and  focus  areas  for  the  SoS  LCA,  the  SoS  LCA  Team  next  turned  its 
attention  to  the  evaluation  process  itself.  The  initial  SoS  LCA  process  model  is  shown  in  Pigure  5. 
As  envisioned,  several  analysis  teams  consisting  of  experts  from  the  SoS  LCA  Team  and  repre- 
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sentatives  from  the  government,  LSI,  and  OTPs  were  to  be  formed.  Each  team  would  tackle  a  sin¬ 
gle  focus  area,  examining  the  relevant  facts,  data,  evidence,  and  plans  and  developing  conclu¬ 
sions.  The  results  from  each  analysis  team  would  be  collected,  summarized,  and  reported  on  by 
the  SoS  LCA  Team,  with  a  final  report  that  detailed  findings  and  recommendations  serving  as 
input  to  the  SoS  PDR.  The  SoS  LCA  final  report,  combined  with  results  from  the  SoS  PDR, 
would  provide  the  basis  for  management  decision  making  regarding  any  corrective  actions. 


Figure  5:  The  Initial  SoS  LCA  Process  Model 

Members  of  the  SoS  LCA  Team  divided  the  focus  areas  among  themselves  based  on  their  respec¬ 
tive  backgrounds  and  expertise. 

3.4  Piloting  the  Initial  Process 

Once  the  initial  process  model  was  in  place,  the  next  challenge  was  to  figure  out  the  details  of  its 
execution.  The  team  determined  that  a  small-scale  pilot  would  be  the  best  way  to  refine  the 
process.  The  intent  of  the  pilot  was  to  prove  out  the  process  steps  in  a  single  area  before  attempt¬ 
ing  to  scale  the  process  up  to  the  full  SoS. 

In  some  respects,  planning  the  pilot  was  as  complicated  as  planning  the  process  itself.  The  prin¬ 
ciple  difficulties  were  determining  the  scope  and  timing  of  the  pilot.  In  terms  of  scope,  it  was  cru¬ 
cial  to  limit  the  extent  of  the  pilot  so  that  it  was  executable  in  a  short  timeframe.  At  the  same  time, 
the  scope  needed  to  be  broad  enough  to  be  useful  as  a  test  run  for  the  larger  effort.  No  less  daunt¬ 
ing  was  the  challenge  of  scheduling  the  pilot.  On  a  program  with  so  many  separate  but  interacting 
aspects,  it  seemed  as  if  there  was  always  a  conflicting  event  to  draw  time,  attention,  and  resources 
away  from  the  LCA  pilot.  The  need  to  minimize  disruption  to  ongoing  program  activities  served 
to  complicate  matters  further. 
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Developing  relationships  with  key  program  personnel  turned  out  to  offer  avenues  of  approach  for 
these  challenges.  The  lead  of  the  C4ISR^  Team  volunteered  his  area  of  responsibility  to  partici¬ 
pate  in  the  LCA  pilot  because  he  saw  value  in  having  some  independent  review  of  the  work  being 
performed  by  his  team.  His  area  of  work,  however,  was  enormous.  Thus,  scoping  became  the  first 
order  of  business.  Adjustment  of  the  pilot  scope  to  be  achievable  yet  worthwhile  was  accom¬ 
plished  through  negotiations  with  the  team  lead.  The  agreed-upon  emphasis  centered  on  a  subset 
of  the  10  focus  areas,  coupled  with  the  C4ISR  lead’s  principle  areas  of  concern.  The  timing  of  the 
pilot  was  propitious;  the  C4ISR  Team  had  a  review  cycle  already  planned  for  the  March  2007 
timeframe,  aligning  nicely  with  the  desired  LCA  pilot  time  frame. 

The  pilot  itself  consisted  of  a  few  parts.  The  first  step  was  to  socialize  the  idea  among  the  C4ISR 
Team  members  so  that  expectations  on  both  sides  were  clear.  The  next  step  was  to  gather  data. 
The  SoS  LCA  Team  spent  about  three  weeks  gathering  evidence  from  existing  C4ISR  plans  and 
technical  documents.  The  SoS  LCA  Team  was  very  explicit  that  no  new  documents  were  to  be 
created  for  the  purposes  of  the  pilot. 

One  early  discovery  during  the  pilot  effort  was  that  SoS  LCA  Team  members  were  not  in  full 
agreement  about  the  goals  or  conduct  of  the  pilot.  Thus,  the  pilot  was  an  invaluable  experience 
both  from  the  perspective  of  checking  and  refining  the  process  and  from  the  point  of  view  of  de¬ 
veloping  a  common  mission  understanding  within  the  SoS  LCA  Team. 

Another  revelation  of  the  pilot  process  was  that  despite  efforts  to  clearly  set  expectations,  there 
was  considerable  misunderstanding  between  the  SoS  LCA  Team,  the  C4ISR  Team,  and  its  vari¬ 
ous  performer  organizations.  Documentation  and  evidence  furnished  by  the  C4ISR  Team  led  to 
more  questions  than  answers,  and  required  several  follow-up  teleconferences  to  request  additional 
documents.  The  documents  provided  by  the  performer  organizations  often  had  timestamps  differ¬ 
ing  by  several  months,  and  frequently  made  what  appeared  to  be  incompatible  assumptions.  Try¬ 
ing  to  resolve  these  inconsistencies  without  interrupting  the  performers’  work  gave  the  SoS  LCA 
Team  an  appreciation  of  the  difficulty  of  performing  a  complete  assessment. 

Actual  evidence  of  feasibility  was  lacking.  That  is  not  to  say  that  the  C4ISR  Team  had  not  done 
its  job,  as  the  contracts  were  unclear  about  what  the  scope  of  “the  job”  of  providing  feasibility 
evidence  was.  Rather,  there  was  little  documentation  to  support  an  independent  conclusion  that 
the  C4ISR  plans  and  approach  were  feasible.  That  is  the  crux  of  the  LCA;  it  relies  upon  evidence 
rather  than  verbal  accounts  and  assurances.  Part  of  the  problem  was  that  the  pilot  was  focused  too 
broadly,  looking  at  several  areas  of  C4ISR  that  had  not  been  planned  for  development  during 
Build  1,  which  is  when  the  pilot  was  executed. 

Despite  these  difficulties,  the  pilot  was  hugely  helpful  in  clarifying  the  SoS  LCA  Team’s  thinking 
about  conducting  the  SoS  LCA.  It  demonstrated  that  program  personnel  would  be  too  busy  to  par¬ 
ticipate  significantly  in  gathering  and  examining  feasibility  evidence.  It  also  demonstrated  the 
need  to  set  expectations  among  developers  and  management  early  and  often  to  ensure  their  buy-in 
to,  and  cooperation  with,  the  process.  The  pilot  also  highlighted  the  need  to  review  emerging  find¬ 
ings  with  stakeholders  to  ensure  accuracy.  The  C4ISR  Team  also  benefitted  from  the  pilot  by  be- 


C4ISR  stands  for  command,  control,  communications,  computers,  intelligence,  surveillance,  and  reconnais¬ 
sance. 
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ing  able  to  raise  issues  that  had  been  of  concern  to  them  for  management  attention  with  the  added 
weight  of  independent  analysis  to  support  them. 


3.5  Refining  the  Process 

As  a  result  of  the  SoS  LCA  pilot  and  other  program  activities,  the  SoS  LCA  Team  made  some 
adjustments  both  to  the  focus  areas  and  to  the  process  model. 

First,  in  consultation  with  program  management,  the  SoS  LCA  Team  modified  the  focus  areas  to 
represent  the  essential  FCS  capabilities,  the  "non-disposable"  elements  of  the  program.  These 
elements  were  ones  that  had  to  work  in  order  to  ensure  that  the  program  produced  a  viable  SoS 
capability  for  the  Warfighter. 

Table  1:  Final  SoS  LCA  Focus  Areas 


Focus  Area 

Description 

Fusion  and  Data  Distribution 

Includes: 

Distributed  Information  Management-effectiveness  of  data  com¬ 
munications  among  platforms  and  systems  over  a  network,  analy¬ 
sis  of  data  reliability,  latency,  etc. 

Distributed  Fusion  Management-effectiveness  of  utilizing  several 
sources  to  gather  and  distribute  fusion  data,  including  network 
usage,  data  reliability,  latency,  etc. 

Rapid  Battlespace  Decontt/ct/on-effectiveness  of  providing,  recog¬ 
nizing,  and  utilizing  battlespace  deconfliction  data  passed  among 

Brigade  Combat  Team  (BCT)  platforms 

Quality  of  Service  (QoS) 

Analysis  of  message  prioritization  effectiveness,  delivery  rates, 
methods  of  prioritization,  etc. 

Information  Assurance  (lA)  & 

Security 

Review  of  the  status  of  efforts  to  develop  and  test  FCS  lA  compo¬ 
nents,  and  a  determination  of  the  attainment  of  necessary  func¬ 
tional  capabilities 

Software  Performance 

Analysis  of  the  projected  ability  of  fielded  FCS  software  (e.g.,  algo¬ 
rithms)  to  enable  planned  capabilities  in  a  comprehensive,  end-to- 
end  operational  environment 

Display  Management  and  Task  Au¬ 
tomation/Task  Integration  Networks 
(TINS) 

Analysis  of  the  performance,  usability,  and  automation  capabilities 
demonstrated  in  Warfighter  Machine  Interface  (WMI)  displays 

Key  Battle  Command  Systems 

Includes: 

Network  Management  System  (NMSj-rewew  of  current  status  of 
development  of  the  Network  Management  System  (NMS)  system, 
as  well  as  analysis  of  the  current  state  of  general  network  man¬ 
agement  issues 

Integrated  Computer  System  f/CSJ-assessment  of  the  develop¬ 
ment  and  issues  related  to  the  various  versions  of  the  ICS  system. 
Determination  of  the  ICS'  ability  to  meet  FCS  design  needs 

Centralized  Controller  fCCj-analysis  of  the  current  state  of  the  CC 
and  its  ability  to  meet  necessary  functional  capabilities 
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Focus  Area 

Description 

End-State  Design 

Determination  of  whether  the  end-state  design  will  meet  opera¬ 
tional  needs.  This  includes  identification  of  risks/issues  and  rec¬ 
ommendations  to  mitigate/address  them 

Software  Producibility 

Analysis  of  PCS  software  against  cost  and  schedule  models  with 
acknowledgement  that  the  scope  and  size  of  the  PCS  software 
effort  is  larger  and  more  complex  than  most  other  projects 

Of  particular  note  are  the  last  two  focus  areas:  end-state  design  and  software  producibility.  End- 
state  design  was  a  capstone  technical  focus  area  that  summed  up  the  findings  of  the  other  technic¬ 
al  focus  areas  in  a  way  that  made  the  findings  relevant  for  management  decision-making.  It  was 
added  as  a  focus  area  midway  through  the  analysis  process  when  the  SoS  LCA  Team  realized  that 
the  other  technical  focus  areas  were  yielding  results  at  too  low  a  level  to  support  strategic  deci¬ 
sion-making.  Refer  to  Section  4.3  in  this  report  for  additional  details  about  end-state  design. 
Software  producibility  looked  specifically  at  the  feasibility  of  developing  the  remaining  software 
within  allocated  schedules  and  budgets  based  on  cost/schedule  analysis  of  the  early  builds  and 
also  on  the  relative  risks  that  emerged  from  the  end-state  design  analysis.  These  two  areas  togeth¬ 
er  summarized  the  overall  SoS  LCA  findings  in  terms  of  program  performance,  providing  the  “so- 
whaf  ’  in  a  report  that  was  otherwise  highly  technical.  The  other  focus  areas  formed  the  detailed 
backup  in  the  final  report  to  management. 


Figure  6  shows  the  changes  to  the  process  model,  which  were  encapsulated  entirely  within  analy¬ 
sis  team  formation  (i.e.,  inside  the  red  oval  denoted  by  the  gray  arrow  on  the  left-hand  side  of  the 
diagram).  The  realities  of  personnel  availability  necessitated  these  slight  adjustments. 


Figure  6:  SoS  LCA  Process  Model  Showing  Refinements 
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As  shown  in  Figure  6,  instead  of  several  analysis  teams,  there  was  only  one  analysis  team  com¬ 
posed  entirely  of  technical  experts  from  the  Software  Engineering  Institute,  the  University  of 
Southern  California,  and  the  Fraunhofer  Center  for  Experimental  Software  Engineering  at  the 
University  of  Maryland.  Rather  than  providing  full-fledged  analysis  team  members,  the  Army  was 
able  only  to  provide  guidance  while  the  LSI  was  only  able  to  facilitate  access  to  documentation. 
OTPs  were  unable  to  participate  in  the  analysis  process  at  all,  although  they  were  called  upon  to 
provide  data  and,  occasionally,  clarification  of  the  data.  Focus  areas  were  assigned  to  one  or  two 
team  members  who  were  then  responsible  for  identifying  the  types  of  artifacts  needed  for  evalua¬ 
tion,  reviewing  them,  and  rendering  recommendations  based  on  review  findings.  The  entire  analy¬ 
sis  team  would  be  responsible  for  reviewing  results  as  they  emerged. 

3.6  Scheduling  the  SoS  LCA 

The  final  step  in  defining  the  SoS  LCA  process  was  determining  where  to  insert  it  into  the  overall 
program  schedule.  The  SoS  LCA  was  not  part  of  the  original  program  plan,  so  it  was  not  reflected 
in  the  program’s  master  plan  or  master  schedule.  For  a  typical  single-system  LCA,  the  execution 
point  is  nominally  at  the  gateway  between  the  elaboration  and  construction  phases  of  the  program. 
For  a  complex  SoS  program,  such  a  distinction  is  less  meaningful,  as  the  various  elements  of  the 
program  move  between  those  phases  at  different  points  in  time.  Thus,  the  ECS  program  had  to 
find  a  schedule  point  that  was  the  logical  equivalent  of  the  gateway  to  the  construction  phase,  but 
at  the  SoS  level. 

The  answer  was  to  conduct  the  SoS  LCA  prior  to  the  program  PDR,  the  point  at  which  the  overall 
SoS  design  would  have  reached  a  sufficiently  mature,  but  not  yet  fully  detailed,  state.  This  loca¬ 
tion  seemed  to  be  a  “natural”  fit  for  the  SoS  LCA  in  the  FCS  schedule.  Tying  the  SoS  LCA  to  the 
program  PDR  also  provided  a  very  convenient  opportunity  to  present  the  results  to  program  man¬ 
agement  in  context  with  other  SoS  technical  details.  Further,  the  program  PDR  needed  a  compo¬ 
nent  that  addressed  software  specifically,  as  preparations  had  focused  almost  exclusively  on  SoS 
hardware  and  external  interaction  concerns.  As  a  practical  matter,  completion  of  the  SoS  LCA  and 
delivery  of  the  final  report  were  added  to  the  conditions  for  successful  closeout  of  the  FCS  pro¬ 
gram  PDR. 
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4  Performing  the  SoS  LCA  Evaluations 


Having  established  a  fixed  endpoint  for  completion,  the  SoS  LCA  Team  worked  backwards  to 
schedule  the  technical  evaluations,  allowing  time  at  the  end  to  draft  a  final  report  to  management. 
To  build  program  management  confidence  that  the  SoS  LCA  would  be  executable,  the  team  de¬ 
cided  on  a  two-phased  execution  approach:  a  Preliminary  Analysis  phase  and  a  Detailed  Analysis 
phase. 

4.1  Preliminary  Analysis 

The  Preliminary  Analysis  phase  examined  the  focus  areas  where  data  was  believed  to  be  most 
available.  This  phase  was  designed  to  give  the  SoS  LCA  Team  insight  into  the  quantity  and  quali¬ 
ty  of  data  available  for  analysis,  to  give  program  leadership  sufficient  opportunity  to  provide 
feedback  on  the  execution  of  the  SoS  LCA,  and  to  allow  the  OTPs  and  LSI  early  insights  into  the 
potential  need  for  corrective  action.  It  also  yielded  insights  to  the  SoS  LCA  Team  into  the  ongo¬ 
ing  evaluation.  For  instance,  the  SoS  LCA  Team  realized  that  some  minor  focus  areas  could  be 
combined  into  meta-focus  areas  (such  as  was  the  case  for  Fusion  and  Data  Distribution),  some 
focus  areas,  such  as  Display  Management  and  Task  Automation/TINs,  needed  no  further  evalua¬ 
tion  during  in  the  Detailed  Analysis  phase,  and  one  focus  area,  namely  Software  Performance, 
was  missing  from  the  plan. 

4.2  Detailed  Analysis 

The  Detailed  Analysis  phase  expanded  upon  the  preliminary  analyses  to  explore  each  focus  area 
in  depth.  The  goal  was  to  identify  specific  strengths  and  weaknesses  in  each  focus  area  along  with 
specific,  actionable  recommendations  for  corrective  action  where  needed.  The  SoS  LCA  Team 
also  met  regularly  to  discuss  emerging  findings  since  the  results  from  one  focus  area  often  had 
bearing  on  other  focus  areas. 

In  addition  to  findings  within  each  focus  area,  at  the  conclusion  of  this  phase,  the  SoS  LCA  Team 
formed  summary  recommendations.  These  recommendations  were  deemed  major,  meaning  that 
there  would  be  significant  benefit  to  the  program  when  implemented,  or  common,  meaning  that 
the  recommendations  were  applicable  to  more  than  one  focus  area. 

4.3  Technical  Capstone:  End-State  Design  Analysis 

The  capstone  technical  focus  area  was  an  examination  of  the  overall  “end-state”  design,  that  is, 
the  software  design  as  it  was  expected  to  exist  at  the  completion  of  the  development  program.  The 
focus  area  sought  to  determine  if  the  end-state  software  design  would  truly  meet  operational  needs 
based  upon  the  available  data,  documentation,  and  plans.  The  analysis  was  an  ambitious  undertak¬ 
ing;  there  had  never  before  been  an  attempt  to  understand  the  SoS  software  design  to  such  a 
depth.  Two  problems  popped  up  immediately:  1)  how  to  characterize  operational  needs  in  terms 
of  software  and  2)  how  to  analyze  a  still-evolving  SoS  software  design. 

Rather  than  figuring  out  how  to  put  operational  needs  in  software  terms,  a  more  logical  approach 
to  the  first  problem  was  to  analyze  the  software  contributions  to  the  definitive  characterization  of 


17  I  CMU/SEI-2009-TR-023 

Approved  for  public  release;  distribution  is  unlimited.  Case  09-91 66.  1 4  January  201 0 


the  program’s  operational  needs — the  SoS  key  performance  parameters  (KPPs).®  With  that  reali¬ 
zation,  the  analysis  approach  then  became  obvious.  The  SoS  LCA  Team  elected  to  use  assurance 
cases  to  analyze  the  SoS  end-state  software  design  with  respect  to  the  KPPs. 


An  assurance  case,  simply,  is  a  structured  set  of  arguments  supported  by  a  body  of  evidence  that 
justifies  belief  that  a  given  claim  is  so.  To  make  one’s  “case,”  one  argues  that  certain  evidence 
supports  a  given  claim.  As  a  practical  matter,  one  constructs  an  assurance  case  by  starting  with  an 
overarching  claim  and  then  iteratively  decomposing  it  into  constituent  claims,  which,  at  the  lowest 
level,  are  supported  by  direct  evidence.  Figure  7  depicts  the  concept. 


Figure  7:  The  Basic  Structure  of  an  Assurance  Case 


The  logic  is  straightforward.  If  the  evidence  is  sufficient,  then  belief  in  sub-claims  1,  3,  and  4  is 
justified.  Belief  in  sub-claims  3,  4  justifies  belief  in  sub-claim  2,  which,  combined  with  belief  in 
sub-claim  1,  justifies  belief  in  the  top-level  overall  claim.^ 

For  FCS,  then,  the  end-state  design  analysis  approach  was  to  use  each  of  the  program’s  KPPs  as  a 
main  claim  and  demonstrate  how  well  the  SoS  software  and  computing  system  designs  supported 
them.  Figure  8  presents  an  oversimplified  example  (note  that  it  is  not  an  actual  FCS  analysis). 


Within  the  U.S.  Department  of  Defense  (DoD),  KPPs  are  an  expression  of  the  system  characteristics  consi¬ 
dered  to  be  essential  for  delivery  of  an  effective  military  capability  [CJCSI  2007].  All  DoD  projects  have  some 
number  of  KPPs  to  satisfy  in  order  to  be  considered  acceptable  from  an  operational  perspective. 

A  more  complete  discussion  on  the  use  of  assurance  cases  for  analyzing  SoS  software  designs  can  be  found 
in  ‘‘Assurance  Cases  for  Design  Analysis  of  Complex  System  of  Systems  Software”  [Blanchette  2009], 
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Figure  8:  Oversimplified  Example  of  an  Assurance  Case  Using  Focus  Area  Findings  as  Evidence 


The  example  shown  in  Figure  8  is  based  on  the  Net  Ready  KPP,  a  common  KPP  among  DoD  sys¬ 
tems  that  must  exchange  information.  The  example  is  drawn  from  the  KPP  definition  found  in  the 
Chairman  of  the  Joint  Chiefs  of  Staff  Instruction  (CJCSI)  6212. OlE  [CJCSI 2008]  (as  shown  hy 
the  context  huhhle  Ctxl)  rather  than  on  a  specific  FCS  program  requirement.  The  figure  shows 
the  main  claim,  Cl,  is  supported  hy  three  sub-claims  C2,  C3,  and  C4.  For  purposes  of  illustration, 
each  sub-claim  is  shown  to  be  directly  supported  by  findings  from  the  SoS  LCA  focus  areas  (Evl, 
Ev2,  and  Ev3).^ 


To  further  illustrate  the  concept,  the  authors  have  arbitrarily  assigned  color-coded  risk  levels  to 
the  evidence  bubbles  as  follows:  red  indicates  a  high  level  of  risk  within  a  focus  area,  yellow  indi¬ 
cates  a  moderate  level  of  risk,  and  green  indicates  a  low  level  of  risk.  The  relative  risk  levels  are 
then  rolled  up  according  to  predetermined  rules  to  establish  a  level  of  confidence  about  the  sup¬ 
ported  claims.  A  sample  rale  set  for  this  example  is  shown  in  Table  2. 


Table  2:  Sample  Rules  for  Rolling  Up  Risk 


Green  (low  risk) 

All  lower  level  claims  and  supporting  evidence  are  green. 

Yellow  (moderate  risk) 

Some  lower  level  claims  and  supporting  evidence  are  a  combination  of  yellow 
and  red. 

Red  (high  risk) 

All,  or  an  overwhelming  majority  of,  lower  level  claims  and  supporting  evidence 
are  red. 

In  this  example,  since  the  findings  for  key  battle  command  systems  indicate  a  high  risk  within  the 
focus  area  (Evl),  the  claim  supported  by  that  evidence  is  also  rated  as  high  risk  (or,  more  correct¬ 
ly,  low  confidence  that  the  claim  is  true).  Since  none  of  the  sub-claims  in  this  example  is  rated 
any  better  than  moderate  risk  (indeed,  C2  is  high  risk),  the  main  claim  Cl  can  be  no  better  than 
moderate  risk.  Hence,  one  would  say  that  there  is  only  moderate  confidence  in  satisfying  the  Net 


In  this  case,  assumption  At  heips  to  justify  the  use  of  the  QoS  focus  area  findings  in  support  of  ciaim  C4. 
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Ready  KPP.  A  full  analysis,  of  course,  would  identify  specific  issues  and  recommendations  at 
each  level.  Armed  with  this  information,  a  program  manager  could  decide  if  the  moderate  risk  of 
not  meeting  the  Net  Ready  KPP  was  acceptable  or,  if  not  acceptable,  decide  on  one  or  more 
courses  of  action  for  reducing  the  risk/increasing  confidence.  Although  this  example  is  deliberate¬ 
ly  negative,  it  should  be  clear  that  strong  evidence  in  support  of  claims  would  lead  one  to  a  high 
level  of  confidence  that  a  given  KPP  was  well  supported. 

For  FCS,  the  use  of  assurance  cases  provided  a  way  to  report  the  technical  findings  from  the  SoS 
LCA  to  program  management  in  a  way  that  related  directly  back  to  program  goals  without  requir¬ 
ing  a  detailed  knowledge  of  software  and  computing  systems.  Each  KPP  had  its  own  assurance 
case,  and  the  findings  from  the  technical  focus  areas  were  used  as  evidence  to  support  one  or 
more  of  those  assurance  cases.  This  approach  tied  both  issues  and  strengths  in  the  software  to  the 
overall  program  goals. 

4.4  Producibility  Analysis 

The  last  of  the  focus  areas  was  concerned  with  programmatic  aspects  of  the  FCS  SoS  software 
development  effort.  Apart  from  technical  feasibility,  the  producibility  analysis  sought  to  demon¬ 
strate  the  feasibility  of  developing  the  SoS  software  and  computing  systems  within  cost  and  sche¬ 
dule  targets. 

Estimating  software  producibility  presented  some  special  challenges.  Unlike  hardware,  which  is 
typically  built  to  completion  before  testing,  software  is  often  implemented  in  a  series  of  incre¬ 
ments  or  builds.  The  software  producibility  costs  for  later  development  increments  tend  to  in¬ 
crease  due  to  previous  increment  breakage  as  well  as  increased  integration  and  test  costs. 

This  breakage  has  several  sources.  One  is  that  errors  discovered  in  earlier  increments  must  be 
fixed  within  the  budgets  and  schedules  of  later  increments,  which  increases  development  work¬ 
load,  while  the  integration  and  test  burden  for  later  increments  is  already  larger  due  to  the  soft¬ 
ware  product  becoming  more  complete.  Others  include  revisions  of  earlier-increment  software  to 
support  later-increment  needs  or  changes,  and  adaptation  of  the  software  to  changes  in  COTS  or 
external-software  interfaces. 

The  magnitude  of  these  Incremental  Development  Productivity  Decline  (IDPD)®  effects  varies 
due  to  several  factors,  such  as  the  degree  of  coupling  to  other  parts  of  the  software  or  the  rate  of 
requirements  or  interface  changes.  Thus,  using  constant-productivity  estimates  for  future  software 
increment  producibility  projections  would  lead  to  severe  underestimation. 

Eurther,  ECS  was  in  a  unique  position  in  that  there  were  no  predecessor  projects  of  equivalent 
scope  and  scale  from  which  to  predict  accurately  its  software  producibility  rates.  The  estimation 
parameters  and  knowledge  bases  of  current  software  estimation  tools  are  generally  good  for  sta¬ 
ble,  standalone,  single -increment  development  of  the  kind  of  software  being  developed  by  the 
ECS  One  Team  Partners.  Flowever,  such  tools  generally  fail  to  account  for  the  degrees  of  program 
and  software  dynamism,  incrementality,  coordination  complexity,  and  system  of  systems  integra- 


IDPD  refers  to  the  phenomenon  in  which  software  deveioper  productivity  tends  to  decrease  during  the  iater 
increments  of  an  incrementai  deveiopment  project  due  to  a  iarger  code  base  that  must  be  maintained  and  in¬ 
tegrated  in  addition  to  producing  new  code  for  the  current  increment  [Boehm  2009]. 
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tion  faced  by  FCS.  These  phenomena  tend  to  decrease  software  productivity  relative  to  the  cost 
model  estimates,  but  it  is  difficult  to  estimate  by  how  much. 

To  calibrate  software  estimates  involves  measurement  of  producibility  data  from  early  increment 
software  deliveries.  For  FCS,  the  best  available  relevant  data  was  from  the  SOSCOE  software 
because  it  underpinned  most  other  software  packages  and  thus  had  several  early  releases.  This  had 
the  benefit  of  an  early  start,  since  it  was  needed  to  support  the  mission  software;  thus,  there  were 
four  increments  upon  which  to  base  the  magnitude  and  evolution  of  its  IDPD  factor.  It  was  not 
fully  representative  of  mission  software,  leaving  uncertainties  about  its  use  to  predict  the  IDPD 
factors  of  the  mission  software  increments.  But  mission  software  from  other  programs  was  not 
representative  of  the  unique  system  of  systems  aspects  of  FCS.  Thus,  data  from  both  sources  were 
used  to  perform  a  sensitivity  analysis  of  the  amount  of  software  that  could  be  produced  within  the 
available  budget  and  schedule.  Continuing  efforts  have  been  underway  to  assess  the  IDPD  factors 
of  various  classes  of  mission  software  as  its  increments  are  produced. 
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5  Results 


Existing  software  reviews  had  specific  artifacts  and  criteria  to  examine,  but  their  scope  was  neces¬ 
sarily  limited.  By  defining  appropriate  and  relevant  criteria  at  the  SoS  level,  the  SoS  LCA  pro¬ 
vided  the  opportunity  to  independently  assess  broader  areas  that  were  critical  to  ECS  success. 

Rooting  results  in  facts  and  data  from  early  software  builds  provided  a  powerful  basis  forjudging 
the  realism  of  plans  for  future  builds,  both  from  technical  and  productivity  perspectives.  Refuta¬ 
tion  of  the  findings  was  possible  if  engineers  were  able  to  supply  evaluators  with  additional  evi¬ 
dence  that  had  not  been  considered^®  and,  indeed,  there  was  one  instance  in  which  evaluation  find¬ 
ings  were  revised  based  on  evidence  that  had  not  previously  been  provided.  Equally  important, 
the  evidence  basis  of  the  overall  analysis  protected  against  pessimistic  appraisals.  In  the  end,  pro¬ 
gram  engineers  were  largely  in  agreement  with  the  SoS  LCA  Team’s  findings. 

Analysis  of  the  producibility  and  end-state  focus  areas  (including  the  roll-ups  of  other  focus  areas) 
indicated  several  areas  where  key  changes  should  be  made  to  the  ECS  SoS  software  development 
effort.  What  is  more,  regular  reports  of  preliminary  findings  helped  management  make  adjust¬ 
ments  to  the  program  or,  in  some  cases,  to  the  in-progress  evaluations.  Gaps  in  design,  architec¬ 
ture,  and  requirements  coverage  that  were  unlikely  to  have  been  uncovered  through  other  types  of 
reviews  were  identified  and  prioritized  for  correction,  providing  a  path  to  reach  necessary  capabil¬ 
ity  levels  in  time  to  support  a  Milestone  C  decision.  Equally  important,  the  SoS  LCA  provided  a 
mechanism  to  report  technical,  cost,  and  schedule  risks,  clearly  tied  to  overall  program  goals,  at 
an  appropriate  level  of  detail  for  senior  program  management,  thereby  enabling  their  understand¬ 
ing  and  supporting  their  decision-making  processes. 

The  ECS  LCA  experience  has  been  extremely  valuable  in  providing  insights  into  processes  better 
fitted  to  the  DoD’s  future  systems  of  systems  programs.  For  example,  it  has  stimulated  several 
improvements  to  the  DoD  version  of  the  Incremental  Commitment  Model  from  its  initial  defini¬ 
tion  as  a  process  model  for  such  future  DoD  human-intensive  systems  of  systems  in  [Pew  2007]. 
The  resulting  upgrade  expressed  in  its  draft  DoD  version  in  [Boehm  2008]  changes  the  pre¬ 
development  phase  name  from  Architecting  to  Eoundations  (including  plans,  budgets,  schedules, 
and  contract  provisions  as  well  as  architecture),  and  changes  the  name  of  its  phase  end-gate  from 
Lifecycle  Architecture  to  Development  Commitment  Review.  It  also  stimulated  extension  of  the 
process  to  include  guidance  for  planning,  executing,  and  eamed-value  managing  the  development 
of  feasibility  evidence  as  a  first-class  deliverable.  Complementary  process  improvement  initia¬ 
tives  for  systems  of  systems  are  underway  at  the  SEI. 


Note  that  such  evidence  had  to  have  been  in  existence  at  the  time  of  the  evaluation,  not  hurriedly  cobbled 
together  after  the  fact  to  defend  against  the  findings. 
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6  Some  Lessons  Learned 


Ideally,  the  SoS  LCA  should  have  been  part  of  program  planning  from  the  outset.  However,  since 
the  concept  was  not  fully  built  into  contract  provisions  and  occurred  after  software  development 
activities  began,  such  was  not  an  option  in  the  FCS  case.  The  effort  to  develop  the  process, 
achieve  buy-in  from  stakeholders,  and  execute  the  analyses  would  have  been  greatly  reduced  had 
the  necessary  contract  provisions  and  resources,  such  as  requirements  to  conduct  an  SoS  LCA  and 
explicit  definition  of  feasibility  evidence  expected  as  an  output  of  the  engineering  work,  been 
built  in  from  the  outset.  Such  resources  are  needed  both  for  the  developers  to  produce  feasibility 
evidence  and  for  the  independent  evaluators  to  assess  it. 

Obtaining  contractual  commitment  to  have  the  feasibility  evidence  produced  and  the  independent 
evaluation  performed  is  essential.  Without  such  commitment,  the  SoS  LCA  Team’s  requests  for 
data  simply  did  not  have  sufficient  weight  when  balanced  against  other  program  priorities.  Once 
the  SoS  LCA  became  a  contractual  requirement,  it  was  easier  for  program  management  at  all  le¬ 
vels  to  prioritize  requests  for  information  appropriately. 

Starting  the  SoS  LCA  process  definition  too  early  is  a  definite  danger.  On  FCS,  the  SoS  LCA 
Team  started  work  before  a  definite  date  for  the  SoS  LCA  had  been  determined.  Due  to  this  float¬ 
ing  schedule,  and  given  all  the  other  program  priorities,  it  was  difficult  for  the  SoS  LCA  Team  to 
maintain  focus  and  commitment.  Even  tying  the  SoS  LCA  to  the  program  PDR  was  not  sufficient 
to  build  team  focus;  initially,  the  planned  date  of  the  PDR  was  too  far  in  the  future  for  team  mem¬ 
bers  to  expend  any  serious  energy  on  defining  or  executing  the  SoS  LCA.  A  period  of  about  a 
year  and  a  half  seems  to  have  been  ideal,  with  about  six  months  for  defining  the  process  and  plan¬ 
ning  the  event  and  12  months  to  execute  the  necessary  process  steps.  The  amount  of  assessment 
effort  would  be  considerably  less  to  the  extent  that  the  developers’  contracts  and  budgets  sup¬ 
ported  development  of  the  feasibility  evidence. 

It  is  vitally  important  to  maintain  close  coordination  within  the  evaluation  team,  especially  when 
that  team  is  geographically  dispersed.  The  SoS  LCA  Team  also  faced  an  additional  coordination 
challenge  in  that  members  came  from  different  organizations  and  thus  had  different  tools  for  vid¬ 
eo  teleconferencing  and  collaboration.  In  such  an  environment,  face-to-face  meetings  were  the 
only  way  to  achieve  effective  brainstorming  sessions,  accomplish  detailed  planning,  and  consoli¬ 
date  evaluation  findings  for  management-level  consumption.  Weekly  teleconferences  are  helpful 
in  keeping  the  effort  moving,  but  the  most  significant  progress  occurred  when  team  members 
were  able  to  meet  in  person. 

A  lesson  from  the  pilot  (and  confirmed  during  the  formal  SoS  LCA)  was  that  it  takes  much  longer 
to  gather  artifacts  than  anyone  had  anticipated.  This  was  true  despite  the  fact  that  all  the  artifacts 
were  already  in  existence.  Part  of  the  difficulty  lay  in  communicating  exactly  which  artifacts  were 
needed  and  which  were  available.  While  the  SoS  LCA  Team  believed  it  communicated  its  needs 
clearly,  differences  in  interpretation  between  the  team  and  program  personnel  providing  the  arti¬ 
facts  caused  confusion  and  slowed  the  process. 

Providing  feedback  to  management  is  essential,  but  it  has  to  be  managed  carefully.  The  initial 
findings  from  the  preliminary  analysis  phase  generated  a  great  deal  of  management  interest.  It 
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also  spurred  more  frequent  updates,  which  in  turn  led  to  an  increase  in  “viewgraph  engineering” 
and  a  corresponding  decrease  in  analysis  due  to  the  time  needed  to  prepare  and  rehearse  briefings 
as  well  as  to  provide  read-ahead  material  in  advance  of  the  actual  update  meetings.  Management 
updates  should  be  scheduled  at  set  points  during  SoS  LCA  execution  as  part  of  the  process. 

The  SoS  LCA  Team  started  out  with  a  rating  scheme  that  was  too  fine.  The  initial  scheme  had 
five  levels,  which  presented  a  problem  when  trying  to  report  early  results  from  the  initial  phase  of 
the  evaluation  to  program  management.  Management  strongly  preferred  so-called  “stop  light 
charts”  showing  green  for  low  risk  areas,  yellow  for  moderate  risk  areas,  and  red  for  high  risk 
areas.  Trying  to  force  the  results  of  a  five -level  rating  system  into  a  three-level  reporting  system 
resulted  in  one  of  two  awkward  solutions:  shades  of  red/yellow/green  or  inconsistency  of  ratings 
within  each  level.  In  either  case,  the  message  became  lost  in  the  medium,  as  more  time  had  to  be 
spent  on  explaining  the  rating  system  than  the  actual  ratings. 

Management  commitment  is  essential  to  achieving  benefit  from  an  SoS  LCA.  The  true  value  of 
the  SoS  LCA  is  in  being  able  to  make  informed  program  decisions  based  on  facts  and  data.  While 
the  intent  is  always  to  conduct  an  SoS  LCA  in  a  way  that  facilitates  decision-making,  without  true 
management  support  the  effort  (like  many  other  program  reviews)  is  at  risk  of  becoming  a  “check 
the  box”  activity  where  the  event  is  held  simply  for  the  sake  of  being  able  to  claim  completion  on 
a  schedule.  Here  again,  contractual  provisions  and  incentives  for  developers  to  plan  for  and  pro¬ 
duce  feasibility  evidence  as  a  first-class  deliverable  are  important  to  obtaining  management  com¬ 
mitment. 

“Socializing”  the  process  among  stakeholders  and  management  in  advance  is  cmcial  to  setting 
expectations.  Many  people  initially  found  the  SoS  LCA  concept  obscure.  Even  when  conducting 
the  pilot  with  an  organization  that  had  been  through  lower  level  LCAs,  there  was  not  a  good  un¬ 
derstanding  of  the  types  of  evidence  sought. 

The  term  “Lifecycle  Architecture”  seems  to  cause  a  great  deal  of  confusion  among  managers  and 
project  personnel  when  applied  in  an  SoS  context.  Particularly,  use  of  the  word  architecture  sets 
incorrect  expectations  that  the  event  will  be  nothing  more  than  an  architectural  review,  which  is 
neither  the  intent  of  an  LCA  at  any  level  nor,  as  explained  earlier,  feasible  at  the  SoS  level  due  to 
the  differing  stages  of  development  among  the  constituent  systems.  Thus,  anyone  planning  to 
adopt  the  SoS  LCA  technique  described  in  this  report  might  wish  to  consider  an  alternate  name 
while  still  maintaining  the  principles  embodied  in  the  LCA  concept. 
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7  Conclusion 


The  SoS  LCA  was  an  effective  means  of  evaluating  the  PCS  SoS  software  and  computing  system 
designs.  Both  the  depth  and  breadth  of  analysis  of  this  software  review  event  far  exceeded  other 
software-specific  review  events  on  the  program.  The  broad,  multi-build  SoS  view,  in  conjunction 
with  the  individual  system  or  individual  software  build  reviews  provided  an  excellent  assessment 
of  the  state  of  the  PCS  software  development  effort.  New  areas  of  analysis  provided  insight  into 
areas  of  the  software  development  program  that  had  never  had  an  in-depth  review. 

Overall,  the  SoS  LCA  was  a  success  both  as  a  new  piece  to  the  PCS  software  review  puzzle  and, 
more  importantly,  in  providing  a  solid  functional  baseline  for  PCS  software  leading  into  an  SoS 
PDR.  The  analyses  not  only  helped  discover  problem  areas,  but  also  recognized  software  pack¬ 
ages  that  were  meeting  or  exceeding  expectations.  This  latter  category  was  particularly  important, 
as  it  provided  guidelines  for  recommended  paths  forward  for  other  software  packages.  Recom¬ 
mended  changes  to  program  processes  and  evaluations  were  proposed  and  being  developed  for 
inclusion  in  the  restructured  program.  Rather  than  mere  action  items  to  be  tracked  to  answer  spe¬ 
cific  questions,  the  SoS  LCA  provided  more  details  and  greater  program  benefit,  including  new 
direction  for  certain  program  areas. 

The  key  element  of  the  SoS  LCA  was  the  ability  to  report  technical,  cost,  and  schedule  risks  at  an 
appropriate  level  of  detail  for  senior  program  management,  thereby  enabling  their  understanding 
and  supporting  their  decision-making  processes. 

The  success  of  the  effort  suggests  possible  follow-up  activities,  including  performing  one  or  more 
“delta”  SoS  LCA  events  to  re-evaluate  areas  where  the  highest  risks  were  found  and  inserting  a 
similar  type  of  effort  before  a  program  CDR  to  gain  deeper  insights  into  the  development  efforts 
at  that  stage  of  the  program.  While  these  specific  actions  have  not  been  tried  on  PCS,  it  seems 
reasonable  to  conclude  that  they  would  be  beneficial.  This  idea  is  consistent  with  the  concept  that 
shortfalls  in  feasibility  evidence  are  uncertainties  and  risks,  and  should  be  covered  by  risk  mitiga¬ 
tion  plans. 

The  PCS  SoS  LCA  activity  has  also  been  a  valuable  learning  experience  for  preparing  and  con¬ 
ducting  future  DoD  SoS  milestone  reviews  and  acquisition  processes. 
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Feedback 


Through  its  Acquisition  Support  Program,  the  SEI  is  working  to  help  improve  the  acquisition  of 
software-intensive  systems  across  the  U.S.  government.  As  part  of  its  mission,  the  SEI  is  very 
interested  in  learning  how  other  organizations  are  tackling  the  challenges  of  developing,  manag¬ 
ing,  or  acquiring  systems  of  systems.  In  addition,  the  SEI  is  pleased  to  discuss  the  information  in 
this  report  in  more  detail. 

Please  send  questions  or  comments  about  this  report  to  Stephen  Blanchette,  Jr. 

(sblanche  @  sei.cmu.edu). 
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Appendix  Acronyms  and  Abbreviations 


The  alphabetical  list  below  contains  all  acronyms,  abbreviations,  and  their  meanings  as  used  in 
this  report. 


ABCS 

Army  Battle  Command  System 

APS 

Active  Protection  System 

ARV-A-L 

Armed  Robotic  Vehicle  -  Assault  -  Light 

B2E 

Build  2  Early 

B2F 

Build  2  Final 

BSE 

Build  S  Early 

BSE 

Build  S  Final 

B4 

Build  4 

BCT 

Brigade  Combat  Team 

BLRIP  DP 

B-Kit  Low  Rate  Initial  Production  Deployment  Period 

C2V 

Command  and  Control  Vehicle 

C4ISR 

Command,  Control,  Communications,  Computers,  Intelli¬ 
gence,  Surveillance,  and  Reconnaissance 

CC 

Centralized  Controller 

CDR 

Critical  Design  Review 

CJCSI 

Chairman  of  the  Joint  Chiefs  of  Staff  Instruction 

CMU 

Carnegie  Mellon  University 

COTS 

commercial  off  the  shelf 
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CSSE 

Center  for  Systems  and  Software  Engineering 

DAB 

Defense  Acquisition  Board 

DCD 

Dismounted  Control  Device 

DoD 

Department  of  Defense 

DRR 

Design  Readiness  Review 

ESC 

Electronic  Systems  Command 

ECS 

Euture  Combat  Systems 

EDTE 

Eorce  Development  Test  and  Experimentation 

EOC 

Eull  Operational  Capability 

ERMV 

ECS  Recovery  and  Maintenance  Vehicle 

ERP 

Eull  Rate  Production 

EY 

Eiscal  Year 

GIG 

Global  Information  Grid 

lA 

Information  Assurance 

ICDR 

Initial  Critical  Design  Review 

ICM 

Incremental  Commitment  Model 

ICS 

Integrated  Computing  System 

ICV 

Infantry  Carrier  Vehicle 

IDPD 

Incremental  Development  Productivity  Decline 

IEEE 

Institute  of  Electrical  and  Electronic  Engineers 
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IMTl 

Integration  Maturity  Xest  I 

IOC 

Initial  Operational  Capability 

lOT&E 

Initial  Operational  Xest  and  Evaluation 

lOTE 

Initial  Operational  Xest  and  Evaluation 

IPDR 

Initial  Preliminary  Design  Review 

IPX 

Integrated  Product  Xeam 

JEEX 

Joint  Expeditionary  Eorce  Experiment 

JROC 

Joint  Requirements  Oversight  Council 

KPP 

Key  Performance  Parameter 

LCA 

Lifecycle  Architecture 

LSI 

Lead  Systems  Integrator 

LUX 

Limited  User  Xest 

MCS 

Mounted  Combat  System 

MSC 

Milestone  C 

MULE-C 

Multi-function  Utility/Logistics  and  Equipment  Vehicle 
Countermine 

MULE-CM 

Multi-function  Utility/Logistics  and  Equipment  Vehicle 
Countermine 

MULE-X 

Multi-function  Utility/Logistics  and  Equipment  Vehicle 
Xransport 

MV 

Medical  Vehicle 

NLOS-C 

Non-Line  of  Sight  Cannon 
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NLOS-LS 

Non-Line  of  Sight  Launch  System 

NLOS-M 

Non-Line  of  Sight  Mortar 

NMS 

Network  Management  System 

OTP 

One  Team  Partner 

P&D 

Production  and  Deployment 

PDR 

Preliminary  Design  Review 

PM 

Program  Manager 

RSV 

Reconnaissance  and  Surveillance  Vehicle 

RUP 

Rational  Unified  Process 

QoS 

Quality  of  Service 

SAIC 

Science  Applications  International  Corporation 

SDD 

System  Development  and  Demonstration 

SEI 

Software  Engineering  Institute 

SO 

Spin  Out 

SoS 

System  Of  Systems 

SOSCOE 

System  of  Systems  Common  Operating  Environment 

SoSER 

System  of  Systems  Eunctional  Review 

SUGV 

Small  Unmanned  Ground  Vehicle 

TINS 

Task  Integration  Networks 

TR 

Technical  Report 
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T-UGS 

Tactical  Unattended  Ground  Sensor 

u.s. 

United  States 

UAV 

Unmanned  Air  Vehicle 

UGS 

Unattended  Ground  Sensor 

UGV 

Unmanned  Ground  Vehicle 

U-UGS 

Urban  Unattended  Ground  Sensor 

WMI 

Warfighter  Machine  Interface 
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